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ABSTRACT 

The construction and validation of a tutor evaluation 
questionnaire are described. The importance of the tutor in 
problem-based learning requires assessment of staff conducting this 
role. After a pilot study in 1990-91 with 100 students and 150 tutors 
who identified important tutor skills, a questionnaire was prepared. 
This questionnaire was tested with six courses in the 1991-92 school 
year, and improvements were made based on the results. The pilot 
studies resulted in a list of 13 Likert scale statements used in an 
instrument administered to about 1,079 students ia 8 courses (142 
tutorial groups) at the Maastricht Medical School (Netherlands) in 
the 1992-93 academic year. Students indicated whether their tutors 
demonstrated the behavior described in each statement. Confirmatory 
factor analysis shows that a 3-factor model comprising 13 items does 
not fit the data, although skipping 2 items improved the model's fit. 
Nevertheless, these two items were considered important for content 
validity. Alpha coefficients indicate that questionnaire reliability 
is high. Findings otherwise indicate- that the students' global and 
analytical ratings about the functioning of the tutor correspond 
highly with each other. Some problems involved in administering the 
questionnaire are discussed. Two tables present study findings. 
(Contains 11 references.) (SLD) 
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Introduction 

The tutor plays a key role in problem-based curricula. This notion is confir- 
' med by research conducted by Gijselaers and Schmidt (1990). These authors 
postulated a causal model of problem-based learning (PBL) to identify and 
measure the effects of important variables, such as group functioning, tutor 
functioning, quality of the problems presented and time spent on self-study. 
One of their findings was that tutor functioning has a direct causal influence 
on the functioning of small-group tutorials which in turn influences students' 
interest in the subject matter. In addition, it has an indirect causal effect on 
student achievement. These results indicate the importance of tutors' abilities 
to guide tutorial groups in an adequate way. 

The importance of the tutor in problem-based curricula requires 
assessment of staff conducting this role. This implies that instruments are 
required to collect information about the functioning of the tutor. This would 
enable the school to provide tutors with feedback. In addition, training and 
remedial teaching should be provided to (prospective) tutors in line with the 
shortcomings pointed out by the evaluation. At the Medical Faculty of the 
University of Limburg, Maastricht, The Netherlands, such an attempt to 
implement a series of those measures has been made. In this paper the 
construction and validation of a tutor evaluation questionnaire will be descri- 
bed. 

Since a tutor evaluation questionnaire should provide teachers with 
feedback, items should reflect key features of the tutor role. This implies that 
the questionnaire needs to be based on the tasks set for the tutor at the 
Medical Faculty of Maastricht, as well as on theoretical conceptions about the 
tutor role, as described in the literature (Barrows, 1988). The role of a tutor in 
a problem-based curriculum mainly consists of guiding the tutorial group. 
The tasks of the tutor are to guide students through all steps of the learning 
process, to encourage students to attain a deeper level of understanding, to 
ensure that all students are involved in the group process, to monitor pro- 
gress of individual students, to motivate students and to help the student 
group to deal with their own problems of interpersonal dynamics (Barrows, 
1988). Coi\sequently, the tutor is a guide and a facilitator and as such an 
important key aspect of problem-based learning. 

Whereas, much descriptive literature is available about the skills a 
tutor should have (Barrows & Tamblyn, 1980; Barrows, 1988), only few 
studies have been conducted to identify important features of the tutor role. 
Most of these studies were concerned with tutor's expertise on subject matter 
under discussion (Davis, Nairn, Paine, Anderson & Oh, 1992; De Voider, 
1982; Feletti, Doyle, Petrovic & Sanson-Fisher, 1982; Moust & Schmidt, 1992). 
De Voider (1982), for imtance, found that tutor's hmctioning as judged by 
students was positively related to expertise on subject matter and experience. 
Feletti et al. (1982) revealed that good tutors were perceived by students as 
having a thorough up-to-date knowledge of the particular problem being 
studied and as encouraging them to revit w their academic progress. Wilker- 
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son (1992) concluded that two factors described the skills seen as most 
helphal by both tutors and students: maintaining positive interactions within 
the group and providing assistance in getting the work of the group 
accomplished. 

Method 

Subjects. Data were collected during eight six-week courses in the academic 
year 1992-1993: two first year courses, two courses in the second year, two 
courses in the third year and two courses in the fourth year. About 1079 
students participated in these eight courses. Both students and tutors were 
randomly assigned to the tutorial groups. Data were aggregated at the 
tutorial group level, since the purpose of the study was to develop an instru- 
ment to provide hitors with feedback about their functioning. The data 
included 142 tutorial groups. 

Instruments. The construction of the questiormaire consisted of several steps. 
The first pilot-study was aimed at identifying important tutor skills. In the 
academic year 1990-1991, 100 students and 150 tutors, divided among the first 
four study years, received a list of 16 items specifying behavioral characteris- 
tics of the tutor. Both students and tutors were asked to rate whether each 
item was assumed to be an important indicator of a tutor's functioning. 
Furthermore, they were asked to indicate whether each item was clearly 
stated. These 16 items were based on descriptions of the tutor role at the 
Maastricht Medical School, on theoretical conceptions about the role of the 
tutor (Barrows, 1988) and on program evaluation instruments used at the 
different faculties of the University of Limburg during the last few years. This 
pilot-study resulted in skipping five items and rewording six items. As a 
consequence, the questionnaire contained eleven items reflecting tutors' 
functioning. In a second pilot-study, in the academic year 1991-1992, this 
questionnaire was used during six courses. The aun of this pilot-study was to 
acquire iiiformation about the way in which tutors should receive individual 
feedback. Students were asked to rate the functioning of the tutor on a Likert- 
scale. Furthermore, students were asked to give an overall judgement (a score 
ranging between 1-10, 6 was "sufficient") for the functioning of the tutor. It 
was assumed that these eleven observed variables are affected by three 
common factors. These factors were related: (1) guiding students through the 
learning process, (2) content knowledge input and (3) commitment to the 
group's leaning. The results of this second pilot-study were reported to 
individual tutors. These tutors were asked to comment on the clarity of the 
items. In general, these tutors considered the feedback as valuable. On the 
basis of these comments and high percentages of students using the "not 
applicable" option for a few items, the questioimaire was improved. These 
pilot-studies resulted in a list of 13 statements, which are affected by the 
three common factors cited above. The factors with their related items are 
shown in Table 1, 
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Table 1 The observed variables and their common factors 



Factor 1 Guiding students through the learning process 

1 The tutor demonstrates to be well-informed about the process of problem- 
based learning 

2 The tutor stimulates all students to participate actively in the tutorial group 
process 

3 The tutor stimulates a careful analysis of the problems 

4 The tutor stimulates the generation of specific learning issues useful for self- 
study 

5 The tutor stimulates an extensive reporting on information collected during 
self-study 

6 The tutor stimulates evaluation of the tutorial group process 
Factor 2 Content knowledge input 

7 The tutor has an understanding of the subject matter covered in the course 

8 The tutor assists students in disting;uishing main issues from minor issues 

9 The tutor uses his or her expert knowledge appropriately 

10 The tutor contributes towards a better understanding of the subject matter 

Factor 3 Commitment to the group's learning 

11 The tutor gives an impression of being motivated 

12 The tutor shows interest in our learning activities during the course 

13 The tutor shows commitment with respect to group functioning 



Students were asked to indicate on a three-point scale whether their tutor 
demonstrated the behavior described in each statement: (1) insufficiently, (2) 
neutral or (3) sufficiently. Besides, a "not applicable" response option was 
added. This option could be selected if students themselves initiated the 
activity described by the statement and tutor intervention in this respect was 
not necessary. In addition, students were asked to give an overall judgement 
(ranging from 1-10, 6 was "sufficient") of the performance of the tutor. 

Statistical analysis, for each item, mean scores, standard deviation and the 
number of missing cases, were computed. The percentage of missmg cases for 
each item were lower than 10 percent. As already mentioned above, the data 
were aggregated at the tutorial group level 

A confirmatory factor analysis was carried out to assess the adequacy of 
the tutor performance model outlmed above. In the confirmatory factor 
model, as specified in this study, all common factors v^ere correlated, 
observed variables 1 through 6 were affected by the first common factor, 
observed variables 7 to 10 were affected by the second common factor, 
variables 11, 12 and 13 by the third common factor. Furthemore, all observed 
variables were assumed to be affected by a unique factor (er or in each 
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variable), and no pairs of unique factors were correlated. The Lisrel VII 
program (Joreskog & Sorbom, 1990) was used to determine whether the data 
confimied this model. In addition, a one-factor model was tested. 

Results 

The number of students rating each tutor varied between 3 and 11. TTie mean 
score on each item for the 142 tutorial groups varied between 218 (SD=.56) 
and 2.82 (SD=.29), shown in Table 2. The low mean score for item 6 may 
reflect tutor's behavior of failing to provide adequate feedback. A study of 
Wilkerson (1992) identifying important skills for the problem-based tutor also 
showed tutors' failure to provide adequate feedback. 

The coefficient alpha for the tutor evaluation questiormaire was .95 (13 
items). The coefficient alpha for factor 1 was .90 (6 items), for factor 2 .94 (4 
items), and for factor 3 .96 (3 items). These results, in general, demonstrate 
that the questionnaire is reliable. 

The validity of the questionnaire was estimated by means of 
confirmatory factor analyses, as already described above. The correlations 
between the observed variables were the input for the confirmatory factor 
analyses (Lisrel VII). The correlation coefficients between the observed 
variables varied between .53 and .90 (N=142), except for item 6 'the tutor 
stimulates the evaluation of the group process' which revealed correlations 
varying between 0.16 and 0.57. The correlations between the common factors 
varied between .67 and .87. Although these coefficients are relatively high, a 
one-factor model did not fit the data. A model is assumed to fit the data if 
three conditions are met: (1) The chi-square divided by the degrees of 
freedom should be lower than 2, a p-value that differs from zero; (2) the root 
mean square residual should be lower than .07; and (3) the goodnessK)f-fit- 
index and the adjusted goodness-of-fit-index, which takes into account the 
number of degrees of freedom, should be higher than .80 (Saris & 
Stronkliorst, 1984). A three-factor model in which observed variables 1 
through 6 are affected by the first common factor, observed variables 7 to 10 
are affected by the second common factor, and variables 11, 12 and 13 are 
affected by the third common factor differed significantly from the data (chi- 
square [62 df] = 160.34, p = .000). The root mean square residual was .057, the 
goodness-of-fit index was .846, and the adjusted goodness-of-fit index was 
774. These results suggest that the first condition specified by Saris and 
Stronkhorst (1984) is not satisfied, whereas both other conditions are satisfied. 
Skipping item 2 and item 6 resulted in a three-factor solution that did not 
differ significantly from the data (chi-square [41 df\ = 74.91, p = .001). The 
root mean square residual of .024, goodness-of-fit index of .906, and adjusted 
goodness-of-fit index o£ .849, suggest that the model fits the data. A four- 
factor model in which the fourth factor consists of item 2 and 6, however, did 
not fit the data. In order to further cross-validate the proposed models, the 
data-set was split up in two sets. Set 1 consisted of the first courses in study 
year 1 to 4 and set 2 consisted of the sec3nd courses in study year 1 to 4. 
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With regard to the first set. both models did not fit the data; i.e., the first 
condition specified by Saris and Stronkhorst (1984) was not satisfied, whereas 
both other conditions are satisfied. With respect to the second set, on the 
* other hand, both models fitted the data. The 11-item model produced the 
following results: chi-square [41 df] = 56.37, p = .055, a root mean square 
residual of .031, a goodness-of-fit index of .871, and adjusted goodness-of-fit 
index of .793. The 13-item model produced the following results: chi-square 
[62 df\ = 104.57, p = .001), a root mean square residual of .050, goodness-of-fit 
index of .812, and adjusted goodness-of-fit index of .724. Since all three 
conditions were satisfied, both models show a reasonable fit with the data. 

Table 2 The observed variables (mean (scale 1-3), standard deviation (SD), 
N=142) and their common factors 



Mean SD 

Factor 1 Guiding students through the learning process 

1 The tutor demonstrates to be well-informed about 

the process of problem-based learning 2.82 .29 

2 The tutor stimulates all students to participate actively 

in the tutorial group process 2.39 .48 

3 The tutor stimulates a careful analysis of the problems 2.62 .42 

4 The tutor stimulates the generation of specific learning 

issues use^ for self-study 2.57 .36 

5 The tutor stimulates an extensive reporting on 

information collected during self-study 2.50 .42 

6 The tutor stimulates evaluation of the tutorial group 
process 



2.18 .56 



Factor 2 Content knowledge input 

7 The tutor has an understanding of the subject matter 

covered in the course 2.70 .39 

8 The tutor assists students in distinguishing main issues 
from minor issues 

9 The tutor uses his or her expert knowledge appropriately 2.60 .46 

10 The tutor contributes towards a better understanding 

of the subject matter 2.57 .45 

Factor 3 Commitment to the group's learning 

11 The tutor gives an impression of being motivated 2.67 .43 

12 The tutor shows interest in our learning activities during 

the course '^^ 

13 The tutor shows conamitment with respect to group 

functioning ^'^^ 
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The overall judgement about the functioning of the tutor (a score 
ranging from 1-10, 6 was "sufficient") correlates highly with all 13 items (all 
were above .73, < .001), with the exception of item 6 "the tutor stimulates 
evaluation of the group process" which correlated .40 (p < .001). 

Since it is assumed that tutor functioning is a key-aspect of a problem- 
based curriculum, a relationship between a tutor's functioning during a 
course and student achievement on a corresponding end-of-course 
examination would be expected. The correlation coefficient between the 
average score on the 13 items of the tutor evaluation questionnaire and the 
average percentage of test items correctly answered, however, was very low 
.08 (n.s., N=142). For separate courses, this correlation coefficient varied 
between -.19 (n.s., n=18) and .46 (n.s., n=18). 

Discussion 

The purpose of this study was to investigate the validity of the tutor 
evaluation questionnaire. The results of the confirmatory factor analyses 
showed that a three-factor-model comprising 13 items does not fit the data, 
since one out of three conditions was not satisfied. Applying the same model 
to a smaller data set, however, revealed that the model fitted the data. 
Although skipping item 2 and 6, respectively "the tutor stimulates all 
students to participate actively in the tutorial group process" and "the tutor 
stimulates evaluation of the tutorial group process" improved the model's fit, 
the 13 item model was nevertheless assumed to be the best model. Item 2 and 
6 reflect important skills a tutor should possess. Skipping these items would 
be detrimental to the content validity of the tutor evaluation questionnaire. 

Alpha coefficients indicated that the reliability of the questiormaire was 
high. Nevertheless, further analyses should be conducted to assess the 
instrument's reliability. The different sources of error caused by students and 
items and the reproducibility of the scores as a function of the number of 
student responses should be estimated. 

An overall judgement at: -mt the functioning of the tutor (expressed as a 
score from 1-10) correlated highly with all 13 items, with the exception of 
item 6 "the tutor stimulates evaluation of the group process". These findings 
indicate that students' global and analytical ratings about the functioning of 
their tutor highly correspond with each other. 

Since it is assumed that tutor functioning is a key-aspect of a problem- 
based curriculum, the relationship between a tutor's functioning dviring a 
course and student achievement on a corresponding end-of-<:ourse 
examination was assessed. The correlation coefficient between the average 
score on the 13 items of the tutor evaluation questionnaire and the average 
percentage of test items, correctly answered was low. A possible explanation 
for this low correlation could be that the tutor has no direct influence on 
achievement. Student achievement seems to be higWy influenced by the 
extent to which the subject matter covered in a course links up with students' 
prior knowledge and time spent on self- study (Gijselaers & Schmidt, 1990). 
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Whereas, most tutors at the Maastricht Medical School highly 
appreciated the practical usefulness of the questionnaire, the implementation 
of the questiormaire showed several bottle-necks. Most problems encountered 
■ were related to the reliability of the data collected. For example, 16 tutors, U 
percent, were rated by less than six students out of a group of eight to eleven 
students. Taken into account that some tutors are rated by only a few 
students, caution is needed in interpreting the results. Especially, if student 
ratings of teaching effectiveness represent a source of information used in 
promotion, tenure and salary decisions. Another problem encountered when 
implementing the tutor evaluation questionnaire was that some students 
rated their tutor although they attended only a few tutorial group meetings. 
It was decided that the ratings of students who visited less than 80 percent of 
the tutorial group meetings were removed. 

Implementing a tutor evaluation questionnaire requires an organizational 
context in which there are opportunities to communicate directly about the 
results. At the Maastricht Medical School the results of the tutor evaluation 
questiormaire are reported to individual tutors, the course-coordinator, and to 
the Educational Coordination Committee. The course-coordinator is 
responsible for the management of the educational design, implementation 
and evaluation of a course. If a tutor scores low, the course-coordinator 
discusses these results with the individual tutor. The results are also 
discussed during the meetings of the Educational Coordination Committee, 
responsible for appointing faculty members to educational roles. If a tutor 
scores low during three subsequent periods of tutoring, the chair of the 
department in which the tutor is working will be informed about the results. 
Such a tutor will be disqualified for several roles, such as the tutor role, the 
role of member of the plarming group, the role of course-coordinator. In 
addition, this negative judgement may influence promotion and tenure 
decisions. The chair of a department will also be informed about consistently 
"good" functioning tutors. This judgement may positively influence promotion 
decisions. The direct communication and reward system should g;uarantee 
that the results are indeed used to improve the functioiung of tutors. A 
review of the literature by Leviton & Hughes (1981) demonstrated five 
general factors influencing the use of evaluation results, such as the relevance 
of the results to the needs of the user, close and direct conununication among 
evaluator and consumers, clear presentation of results, credibility of the 
results which is narrowly related to users' preconceptions about the quality of 
the results and the consensus between research findings and information 
from other sources. During pilot-studies, as described earlier, a direct 
communication between evaluator and consumers took place, in which the 
relevance of the results was discussed as well as the clarity of the 
presentation of the individual feedback each tutor receives. 

Providing tiJtors with feedback about their functioning should of course 
imply that training programs are offered to tutors scoring very low, since 
improvement of tutor behavior is the ultimate purpose of collecting these . , 

data. At the Maasbricht Medical Faculty all tutors are obliged to attend a tutor 

■ ■ vy*, 
• : -.•••-=^■^■• 
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training before they may guide a tutorial group. During two day", tutors are 
prepared for the tutor role and opportunities are offered to practice the 
required skills. However, remedial teaching activities providing practice and 
feedback should also be offered, since a one-time tutor trairung program will 
be not be sufficient for some tutors. These activities could, for example, 
include observing a good functioning tutor in a tutorial group. Besides, 
training should be offered in which tutors are observed in simulated groups 
and receive direct feedback about their functioning. Presently, initiatives are 
undertaken to realize these activities. 
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